数据 | 核心生产要素[1]数据工程03.处理与计算[1]技术栈

Flink-核心原理

一、

二、状态计算

三、容错机制

四、资源信息

队列、算子并行度（32）

TM数（16）、TM内存（4G）、JM内存（4G）

「模式」

大流量作业：独立 Application Cluster

小作业：合并到 Session Cluster 共享 TM

「解释」

物理机（Node）、Kubernetes Pod、Flink Job、JobManager / TaskManager、Task / SubTask
之间的 层级关系 & 调度关系

1.物理机（Node）：Flink 在 K8s 中运行，一个Node可以跑多个 TaskManager Pods，多个JobManager Pod
2.Pod 层（K8s 调度单位）：

「Flink on K8s」

flowchart TB

    %% =========== Kubernetes Node ===========
    subgraph Node[物理节点 / Kubernetes Node]
        direction TB

        JM_Pod[Pod：JobManager Pod]

        TM_Pod1[Pod：TaskManager Pod 1]
        TM_Pod2[Pod：TaskManager Pod 2]
    end

    %% =========== Pod 内容 ===========
    JM_Pod --> JM[JobManager]

    TM_Pod1 --> TM1[TaskManager 1]
    TM_Pod2 --> TM2[TaskManager 2]


    %% =========== Slot 层 ===========
    subgraph SL1[TaskManager 1 Slots]
        direction TB
        TM1_S1[Slot 1]
        TM1_S2[Slot 2]
        TM1_S3[Slot 3]
    end

    subgraph SL2[TaskManager 2 Slots]
        direction TB
        TM2_S1[Slot 1]
        TM2_S2[Slot 2]
        TM2_S3[Slot 3]
    end

    TM1 --> SL1
    TM2 --> SL2


    %% =========== Flink Jobs ===========

    subgraph JobA[Flink Job A]
        direction TB
        A_Task1[Task A1]
        A_Task2[Task A2]
    end

    subgraph JobB[Flink Job B]
        direction TB
        B_Task1[Task B1]
        B_Task2[Task B2]
    end

    subgraph JobC[Flink Job C]
        direction TB
        C_Task1[Task C1]
        C_Task2[Task C2]
    end


    %% =========== SubTasks ===========

    subgraph SubTasks[Tasks 并行实例（SubTasks）]
        direction TB
        A1[SubTask A1-1]
        A2[SubTask A1-2]
        A3[SubTask A2-1]
        A4[SubTask A2-2]

        B1[SubTask B1-1]
        B2[SubTask B1-2]
        B3[SubTask B2-1]
        B4[SubTask B2-2]

        C1[SubTask C1-1]
        C2[SubTask C1-2]
        C3[SubTask C2-1]
        C4[SubTask C2-2]
    end


    %% =========== 调度关系 ===========
    JM --> JobA
    JM --> JobB
    JM --> JobC

    A_Task1 --> A1
    A_Task1 --> A2
    A_Task2 --> A3
    A_Task2 --> A4

    B_Task1 --> B1
    B_Task1 --> B2
    B_Task2 --> B3
    B_Task2 --> B4

    C_Task1 --> C1
    C_Task1 --> C2
    C_Task2 --> C3
    C_Task2 --> C4


    %% ========== 多 Job 共享 Slot ==========
    A1 --> TM1_S1
    A2 --> TM2_S1
    A3 --> TM1_S2
    A4 --> TM2_S2

    B1 --> TM1_S3
    B2 --> TM2_S3
    B3 --> TM1_S1
    B4 --> TM2_S2

    C1 --> TM1_S2
    C2 --> TM2_S1
    C3 --> TM1_S3
    C4 --> TM2_S3

「Flink 集群（Session Cluster）完整架构图」

flowchart TB

%% ========================================================
%%  Kubernetes 节点层（真实实体：实线）
%% ========================================================
subgraph Node1[Node 1（物理机/虚机）]
    direction TB
    subgraph JM_Pod[JobManager Pod]
        direction TB
        JM_Container[Container]
        JM[JobManager（JVM 进程
Dispatcher / REST / Scheduler）]
    end

    subgraph TM_Pod_1[TaskManager Pod #1]
        direction TB
        TM_Container_1[Container]
        subgraph TM1[TaskManager #1]
            direction TB
            Slot11[Slot #1]
            Slot12[Slot #2]
        end
    end
end

subgraph Node2[Node 2]
    direction TB
    subgraph TM_Pod_2[TaskManager Pod #2]
        direction TB
        TM_Container_2[Container]
        subgraph TM2[TaskManager #2]
            direction TB
            Slot21[Slot #1]
            Slot22[Slot #2]
        end
    end
end

subgraph Node3[Node 3]
    direction TB
    subgraph TM_Pod_3[TaskManager Pod #3]
        direction TB
        TM_Container_3[Container]
        subgraph TM3[TaskManager #3]
            direction TB
            Slot31[Slot #1]
            Slot32[Slot #2]
        end
    end
end


%% ========================================================
%%   Flink Job 逻辑层（虚线）
%% ========================================================
classDef dashed stroke-dasharray: 5 5;

subgraph JobGraph["Flink Job（抽象逻辑）"]
    class JobGraph dashed;

    direction TB
    subgraph TaskA["Task A（Stage 1）"]
        class TaskA dashed;
        A1["SubTask A-1"]
        A2["SubTask A-2"]
        A3["SubTask A-3"]
    end

    subgraph TaskB["Task B（Stage 2）"]
        class TaskB dashed;
        B1["SubTask B-1"]
        B2["SubTask B-2"]
        B3["SubTask B-3"]
    end
end


%% ========================================================
%%   调度关系（虚线箭头：逻辑 → 实体 Slot）
%% ========================================================
A1 -.-> Slot11
A2 -.-> Slot21
A3 -.-> Slot31

B1 -.-> Slot12
B2 -.-> Slot22
B3 -.-> Slot32

「Session Cluster vs Application Cluster 架构对比图」

flowchart TB

subgraph Session["Flink Session Cluster"]
    direction TB

    subgraph SessionJM["JobManager Pod（长期存在）"]
        SJMgr[JobManager]
    end

    subgraph SessionTM["多个 TaskManager Pod（所有 Job 共享）"]
        STM1[TaskManager Pod #1]
        STM2[TaskManager Pod #2]
        STM3[TaskManager Pod #3]
    end

    SJ1["Client 提交多个 Job"] --> SJMgr
    SJMgr --> STM1
    SJMgr --> STM2
    SJMgr --> STM3

    SJNote["特点：
• 多个 Job 共享 JM/TM
• 集群先启动，Job 后进入
• 资源抢占（不隔离）
• 适合小作业/交互式环境"]
end


subgraph Application["Flink Application Cluster"]
    direction TB

    APClient["用户提交：flink run-application"]

    subgraph AppJM["JobManager Pod（随 Job 启动/销毁）"]
        AJMgr[JobManager（仅运行该 Job）]
    end

    subgraph AppTM["TaskManager Pod（仅该 Job 使用）"]
        ATM1[TaskManager Pod #1]
        ATM2[TaskManager Pod #2]
    end

    APClient --> AJMgr
    AJMgr --> ATM1
    AJMgr --> ATM2

    APNote["特点：
• 每个 Job 生成独立集群
• Job 结束 → 集群销毁
• 强隔离，不共享 TM
• 适合生产大 Job / 长任务"]
end
flowchart TB
    %% =========================
    %% Title
    %% =========================
    A[三种 Flink 部署模式对比
Session vs Application vs Per-Job]

    %% =========================
    %% Session Cluster
    %% =========================
    subgraph S[Session Cluster（长期运行，可共享资源）]
        direction TB

        S-JM[JobManager Pod
集群长期存在]

        subgraph S-TM-POOL[TaskManager Pods 池
多个作业共享]
            S-TM1[TM Pod 1
Slots: 1..N]
            S-TM2[TM Pod 2
Slots: 1..N]
            S-TM3[TM Pod 3
Slots: 1..N]
        end

        subgraph S-JOBS[多个作业同时运行]
            S-JOB1[Job-1]
            S-JOB2[Job-2]
            S-JOB3[Job-3]
        end

        S-JOB1 --> S-TM1
        S-JOB1 --> S-TM2

        S-JOB2 --> S-TM2
        S-JOB2 --> S-TM3

        S-JOB3 --> S-TM1
        S-JOB3 --> S-TM3

        S-JOBS --> S-JM
    end

    %% =========================
    %% Application Cluster
    %% =========================
    subgraph A2[Application Cluster（每个 APP 一个集群）]
        direction TB

        A2-JM[JobManager Pod
随 App 启动，App 结束即删除]

        subgraph A2-TM-PODS[TaskManager Pods（仅此 App 使用）]
            A2-TM1[TM Pod 1
Slots:1..N]
            A2-TM2[TM Pod 2
Slots:1..N]
        end

        subgraph A2-JOBS[App 内多个 Job]
            A2-JOB1[Job A-1]
            A2-JOB2[Job A-2]
        end

        A2-JOB1 --> A2-TM1
        A2-JOB2 --> A2-TM2

        A2-JOBS --> A2-JM
    end

    %% =========================
    %% Per-Job Cluster
    %% =========================
    subgraph P[Per-Job Cluster（每个 Job 独立集群）]
        direction TB

        P-JOB[单个 Job（独占资源）]

        P-JM[JobManager Pod（随 Job 启动）]

        subgraph P-TM-PODS[TaskManager Pods（仅此 Job 使用）]
            P-TM1[TM Pod 1
Slots: 1..N]
            P-TM2[TM Pod 2
Slots: 1..N]
        end

        P-JOB --> P-TM1
        P-JOB --> P-TM2

        P-JOB --> P-JM
    end

    %% Top level links
    A --> S
    A --> A2
    A --> P